Efficient Evolutionary Dynamics with Extensive-Form Games

Evolutionary game theory combines game theory and dynamical systems and is customarily adopted to describe evolutionary dynamics in multi–agent systems. In particular, it has been proven to be a successful tool to describe multi–agent learning dynamics. To the best of our knowledge, we provide in this paper the first replicator dynamics applicable to the sequence form of an extensive–form game, allowing an exponential reduction of time and space w.r.t. the currently adopted replicator dynamics for normal form. Furthermore, our replicator dynamics is realization equivalent to the standard replicator dynamics for normal form. We prove our results for both discrete–time and continuous–time cases. Finally, we extend standard tools to study the stability of a strategy profile to our replicator dynamics. Introduction Game theory provides the most elegant tools to model strategic interaction situations among rational agents. These situations are customarily modeled as games (Fudenberg and Tirole 1991) in which the mechanism describes the rules and strategies describe the behavior of the agents. Furthermore, game theory provides a number of solution concepts. The central one is Nash equilibrium. Game theory assumes agents to be rational and describes “static” equilibrium states. Evolutionary game theory (Cressman 2003) drops the assumption of rationality and assumes agents to be adaptive in the attempt to describe dynamics of evolving populations. Interestingly, there are strict relations between game theory solution concepts and evolutionary game theory steady states, e.g., Nash equilibria are steady states. Evolutionary game theory is commonly adopted to study economic evolving populations (Cai, Niu, and Parsons 2007) and artificial multi–agent systems, e.g., for describing multi–agent learning dynamics (Tuyls, Hoen, and Vanschoenwinkel 2006; Tuyls and Parsons 2007; Panait, Tuyls, and Luke 2008) and as heuristics in algorithms (Kiekintveld, Marecki, and Tambe 2011). In this paper, we develop efficient techniques for evolutionary dynamics with extensive–form games. Extensive–form games are a very important class of games. They provide a richer representation than strategic– form games, the sequential structure of decision–making beCopyright © 2013, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. ing described explicitly and each agent being allowed to be free to change her mind as events unfold. The study of extensive–form games is carried out by translating the game by means of tabular representations (Shoham and LeytonBrown 2008). The most common is the normal form. Its advantage is that all the techniques applicable to strategic– form games can be adopted also with this representation. However, the size of normal form grows exponentially with the size of the game tree, thus being impractical. The agent form is an alternative representation whose size is linear in the size of the game tree, but it makes, even with two agents, each agent’s best–response problem highly non– linear. To circumvent these issues, sequence form was proposed (von Stengel 1996). This form is linear in the size of the game tree and does not introduce non–linearities in the best–response problem. On the other hand, standard techniques for strategic–form games cannot be adopted with such representation, e.g. (Lemke and Howson 1964), thus requiring alternative ad hoc techniques, e.g. (Lemke 1978). In addition, sequence form is more expressive than normal form. For instance, working with sequence form it is possible to find Nash–equilibrium refinements for extensive– form games—perfection based Nash equilibria and sequential equilibrium (Miltersen and Sørensen 2010; Gatti and Iuliano 2011)—while it is not possible with normal form. To the best of our knowledge, there is no result dealing with the adoption of evolutionary game theory tools with sequence form for the study of extensive–form games, all the known results working with the normal form (Cressman 2003). In this paper, we originally explore this topic, providing the following main contributions. • We show that the standard replicator dynamics for normal form cannot be adopted with the sequence form, the strategies produced by replication not being well–defined sequence–form strategies. • We design an ad hoc version of the discrete–time replicator dynamics for sequence form and we show that it is sound, the strategies produced by replication being well– defined sequence–form strategies. • We show that our replicator dynamics is realization equivalent to the standard discrete–time replicator dynamics for normal form and therefore that the two replicator dynamics evolve in the same way. • We extend our discrete–time replicator dynamics to the continuous–time case, showing that the same properties are satisfied and extending standard tools to study the stability of the strategies to our replicator. Game theoretical preliminaries Extensive–form game definition. A perfect–information extensive–form game (Fudenberg and Tirole 1991) is a tuple (N,A,V,T, ι, ρ,χ,u), where: N is the set of agents (i ∈ N denotes a generic agent), A is the set of actions (Ai ⊆ A denotes the set of actions of agent i and a ∈ A denotes a generic action), V is the set of decision nodes (Vi ⊆ V denotes the set of decision nodes of i), T is the set of terminal nodes (w ∈ V ∪ T denotes a generic node and w0 is root node), ι ∶ V → N returns the agent that acts at a given decision node, ρ ∶ V → ℘(A) returns the actions available to agent ι(w) at w, χ ∶ V ×A → V ∪ T assigns the next (decision or terminal) node to each pair ⟨w,a⟩ where a is available at w, and u = (u1, . . . , u∣N ∣) is the set of agents’ utility functions ui ∶ T → R. Games with imperfect information extend those with perfect information, allowing one to capture situations in which some agents cannot observe some actions undertaken by other agents. We denote by Vi,h the h–th information set of agent i. An information set is a set of decision nodes such that when an agent plays at one of such nodes she cannot distinguish the node in which she is playing. For the sake of simplicity, we assume that every information set has a different index h, thus we can univocally identify an information set by h. Furthermore, since the available actions at all nodes w belonging to the same information set h are the same, with abuse of notation, we write ρ(h) in place of ρ(w) with w ∈ Vi,h. An imperfect–information game is a tuple (N,A,V,T, ι, ρ,χ,u,H) where (N,A,V,T, ι, ρ,χ,u) is a perfect–information game and H = (H1, . . . ,H∣N ∣) induces a partition Vi = ⋃h∈Hi Vi,h such that for all w,w ′ ∈ Vi,h we have ρ(w) = ρ(w). We focus on games with perfect recall where each agent recalls all the own previous actions and the ones of the opponents (Fudenberg and Tirole 1991).